Collaborators

Russ Thurow USDA Forest Service Rocky Mountain Research Station

Claire McGrath Natural Resources Specialist, Columbia Hydropower Branch at NOAA Fisheries, West Coast Region,

Kevin See Biometrician, Biomark Inc, Boise, ID,

Goals

Salmon redd counts are widespread method to estimate the number of returning adult spawners. However, despite its prevalence in the Northwest, the reliability of redd counts is unknown. This work is focused on developing a statistical model to estimate the observer error in redd surveys, using a variety of covariates related to the habitat and the observer. We described three types of observer error:

  • omission rate, \(\omega\) (proprotion of redds available to be counted that were missed by the observer)
  • commission rate, \(\eta\) (rate of redds counted by an observer that were not actually redds)
  • net error, \(\gamma\) (ratio of observed redds to true redds). This was modeled using log(net error) as the reponse.

Methods

Possible covariates in each error model are shown in Table 1. To make comparisons with AICc, the random effects must be identical across all models. Therefore, we ensured that the random effect of year was added to any model that didn’t have it.

Table 1: Possible covariates included in each observer error model.

Type Covariate Air Ground
Random Reach X X
Random Surveyor X
Random Year X X
Fixed ANNDist_log X X
Fixed AveAge X X
Fixed AveBadCond X
Fixed AveCanopy X X
Fixed AveContrast X X
Fixed AveDepth X X
Fixed AveOverlap X X
Fixed AveSunny X
Fixed AveWidth X X
Fixed ExperienceCat X
Fixed I(AveDepth^2) X X
Fixed LYabund X X
Fixed LYabund:PeakQ X X
Fixed OthrDens X X
Fixed PeakQ X X
Fixed redd_dens X X
Fixed Slope X X

All covariates were z-scored, and all models were fit using the glmer or lmer functions from the lme4 package (Bates et al. 2015) in R software (R Core Team 2019). The amount of variation explained by fixed and random effects was calculated using the methods of Nakagawa and Schielzeth (2013). Using estimated predictions of the rates for omission (\(\hat{\omega}\)), commission (\(\hat{\eta}\)) and net error (\(\hat{\gamma}\)), we predicted the number of actual redds by either dividing the observed counts, \(c\), by estimates of net error, or by multiplying the observed counts by 1 - estimated rate of commission, and then dividing by 1 - estimated rate of omission.

We performed a cross validation by dividing each survey type data into 5 training datasets where 20% of the data was withheld for testing, and then fitting the naive and best AICc model formulations to the remaining data, and then using those fits to predict the error rates and true number of redds for each survey in the year that had been withheld.

\[ \begin{aligned} redds_{ne} &= \frac{c}{\hat{\gamma}} \\ redds_{om} &= c * \frac{1 - \hat{\eta}}{1 - \hat{\omega}} \end{aligned} \]

The observed error rates are showin in Figure 1.

Figure 1: Observed error rates.

Figure 1: Observed error rates.

Results

Model Coefficients

The model coefficients of the full, best (by AICc) and model averaged models are shown in Table 2.

Table 2: Estimated coefficients for various observer error models.

Survey Resp Covariate avg best full
Ground Com (Intercept) -0.962 -0.961 -0.961
Ground Com ANNDist_log 0.360 0.359 0.359
Ground Com AveAge -0.042 -0.042 -0.042
Ground Com AveCanopy 0.022 0.022 0.022
Ground Com AveContrast 0.018 0.018 0.018
Ground Com AveDepth 0.152 0.152 0.152
Ground Com AveOverlap 0.027 0.027 0.027
Ground Com AveWidth -0.137 -0.137 -0.137
Ground Com ExperienceCat.L -0.162 -0.162 -0.162
Ground Com ExperienceCat.Q 0.305 0.305 0.305
Ground Com I(AveDepth^2) 0.064 0.064 0.064
Ground Com LYabund 0.270 0.270 0.270
Ground Com LYabund:PeakQ 0.107 0.107 0.107
Ground Com OthrDens 0.223 0.223 0.223
Ground Com PeakQ -0.122 -0.122 -0.122
Ground Com redd_dens 0.102 0.102 0.102
Ground Com Slope 0.034 0.034 0.034
Ground Net (Intercept) -0.390 -0.390 -0.236
Ground Net ANNDist_log 0.260 0.258 0.269
Ground Net AveAge -0.059 - -0.059
Ground Net AveCanopy -0.017 - -0.017
Ground Net AveContrast 0.020 - 0.020
Ground Net AveDepth 0.035 - 0.035
Ground Net AveOverlap -0.004 - -0.008
Ground Net AveWidth -0.096 - -0.096
Ground Net ExperienceCat.L 0.322 - 0.322
Ground Net ExperienceCat.Q -0.170 - -0.170
Ground Net I(AveDepth^2) -0.002 - -0.002
Ground Net LYabund 0.263 - 0.263
Ground Net LYabund:PeakQ 0.177 - 0.177
Ground Net OthrDens 0.003 - 0.003
Ground Net PeakQ -0.003 - -0.003
Ground Net redd_dens 0.049 - 0.031
Ground Net Slope 0.024 - 0.024
Ground Omi (Intercept) -0.387 -0.387 -0.387
Ground Omi ANNDist_log -0.069 -0.069 -0.069
Ground Omi AveAge 0.174 0.174 0.174
Ground Omi AveCanopy 0.002 0.002 0.002
Ground Omi AveContrast -0.051 -0.051 -0.051
Ground Omi AveDepth 0.110 0.110 0.110
Ground Omi AveOverlap 0.098 0.098 0.098
Ground Omi AveWidth 0.025 0.025 0.025
Ground Omi ExperienceCat.L -0.746 -0.746 -0.746
Ground Omi ExperienceCat.Q 0.669 0.669 0.669
Ground Omi I(AveDepth^2) -0.049 -0.049 -0.049
Ground Omi LYabund -0.565 -0.565 -0.565
Ground Omi LYabund:PeakQ -0.736 -0.736 -0.736
Ground Omi OthrDens -0.031 -0.031 -0.031
Ground Omi PeakQ -0.233 -0.233 -0.233
Ground Omi redd_dens 0.245 0.244 0.244
Ground Omi Slope 0.380 0.380 0.380
Survey Resp Covariate avg best full
Air Com (Intercept) -1.339 -1.342 -1.285
Air Com ANNDist_log 0.382 0.290 0.296
Air Com AveAge -0.206 - -0.052
Air Com AveBadCond -0.003 - -0.003
Air Com AveCanopy -0.041 - -0.037
Air Com AveContrast -0.060 - -0.007
Air Com AveDepth -0.191 - -0.191
Air Com AveOverlap -0.136 -0.136 -0.143
Air Com AveSunny 0.332 - 0.291
Air Com AveWidth 0.164 - 0.164
Air Com I(AveDepth^2) 0.050 - 0.050
Air Com LYabund 0.730 - 0.747
Air Com LYabund:PeakQ 0.679 - 0.701
Air Com OthrDens 0.069 - 0.072
Air Com PeakQ -0.167 - -0.158
Air Com redd_dens -0.204 -0.198 -0.138
Air Com Slope 0.022 - 0.022
Air Net (Intercept) -0.402 -0.401 -0.215
Air Net ANNDist_log 0.283 0.303 0.238
Air Net AveAge -0.179 - -0.179
Air Net AveBadCond 0.003 - 0.003
Air Net AveCanopy -0.072 - -0.070
Air Net AveContrast 0.077 - 0.038
Air Net AveDepth -0.061 - -0.062
Air Net AveOverlap -0.102 - -0.105
Air Net AveSunny 0.137 - 0.060
Air Net AveWidth -0.040 - -0.041
Air Net I(AveDepth^2) 0.026 - 0.026
Air Net LYabund 0.463 - 0.465
Air Net LYabund:PeakQ 0.445 - 0.447
Air Net OthrDens -0.080 - -0.060
Air Net PeakQ -0.084 - -0.083
Air Net redd_dens 0.010 - -0.051
Air Net Slope -0.052 - -0.052
Air Omi (Intercept) -0.565 -0.565 -0.522
Air Omi ANNDist_log -0.202 - -0.206
Air Omi AveAge 0.507 0.507 0.591
Air Omi AveBadCond -0.012 - -0.012
Air Omi AveCanopy 0.046 - 0.046
Air Omi AveContrast 0.014 0.014 0.010
Air Omi AveDepth 0.374 - 0.374
Air Omi AveOverlap 0.287 0.287 0.180
Air Omi AveSunny -0.050 - -0.050
Air Omi AveWidth -0.096 - -0.096
Air Omi I(AveDepth^2) -0.097 - -0.097
Air Omi LYabund -0.457 - -0.457
Air Omi LYabund:PeakQ -0.732 - -0.732
Air Omi OthrDens 0.108 - 0.108
Air Omi PeakQ 0.157 - 0.157
Air Omi redd_dens -0.085 - -0.091
Air Omi Slope 0.308 - 0.308

Ground Surveys

The relative importance of each covariate in each model is shown in Figure 2, while the amount of the variance explained by fixed and random effects in the best AICc model is shown in Figure 3. Observed versus predicted rate plots are shown in Figures 4, 6 and 8.

Figure 2: Relative importance of each covariate in ground-based observer error models

Figure 2: Relative importance of each covariate in ground-based observer error models

Figure 3: How much variance in the model response is explained by the fixed and random effects in the best AICc model.

Figure 3: How much variance in the model response is explained by the fixed and random effects in the best AICc model.

Omission

Figure 4: Observed versus predicted rates of omission using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 4: Observed versus predicted rates of omission using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 5: Correlations between observed omission rates and three model predictions (model averaged, single best and naive).

Figure 5: Correlations between observed omission rates and three model predictions (model averaged, single best and naive).

Commission

Figure 6: Observed versus predicted rates of commission using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 6: Observed versus predicted rates of commission using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 7: Correlations between observed commission rates and three model predictions (model averaged, single best and naive).

Figure 7: Correlations between observed commission rates and three model predictions (model averaged, single best and naive).

Net Error

Figure 8: Observed versus predicted rates of net error using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 8: Observed versus predicted rates of net error using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 9: Correlations between observed net error rates and three model predictions (model averaged, single best and naive).

Figure 9: Correlations between observed net error rates and three model predictions (model averaged, single best and naive).

Leave-One-Out Cross Validation

Rate Estimates

We examined the bias in estimates rates, using both the best (by AICc) model and the naive model (only random effects).

Redd Estimates

For ground-based surveys, both methods provided fairly unbiased estimates of the true number of redds (Figure 10), although the omission/commision models had slightly higher absolute and relative bias (Table 3).

Figure 10: Boxplots of absolute and relative bias for each type of predictive model.

Figure 10: Boxplots of absolute and relative bias for each type of predictive model.

Table 3: Summary statistics of predictions of total redds from leave-one-out cross validation using the net error and the omission/commission models.

Model Median # Obs. Redds Median # True Redds Median Adjustment Median Abs. Bias Median Rel. Bias (%) RMSE
Best Net Error 36 38 5.3 -0.1 -0.6 24.1
Best Omis / Comm Error 36 38 4.3 -0.6 -2.1 17.7
Naive Net Error 36 38 5.7 0.0 0.1 24.0
Naive Omis / Comm Error 36 38 4.0 0.3 1.0 17.5
Observed 36 38 - -2.0 -8.0 18.9
Figure 11: Observed number of true redds vs. leave-one-out cross validated predicted redds based on either the best AICc or naive versions of the net error or omission/commission models. Dashed line is the 1-1 line, while solid line with gray error ribbon is the best fit linear model to these data.

Figure 11: Observed number of true redds vs. leave-one-out cross validated predicted redds based on either the best AICc or naive versions of the net error or omission/commission models. Dashed line is the 1-1 line, while solid line with gray error ribbon is the best fit linear model to these data.

Air Surveys

The relative importance of each covariate in each model is shown in Figure 12, while the amount of the variance explained by fixed and random effects in the best AICc model is shown in Figure 13. Observed versus predicted rate plots are shown in Figures 14, 16 and 18.

Figure 12: Relative importance of each covariate in ground-based observer error models

Figure 12: Relative importance of each covariate in ground-based observer error models

Figure 13: How much variance in the model response is explained by the fixed and random effects in the best AICc model.

Figure 13: How much variance in the model response is explained by the fixed and random effects in the best AICc model.

Omission

Figure 14: Observed versus predicted rates of omission using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 14: Observed versus predicted rates of omission using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 15: Correlations between observed omission rates and three model predictions (model averaged, single best and naive).

Figure 15: Correlations between observed omission rates and three model predictions (model averaged, single best and naive).

Commission

Figure 16: Observed versus predicted rates of commission using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 16: Observed versus predicted rates of commission using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 17: Correlations between observed commission rates and three model predictions (model averaged, single best and naive).

Figure 17: Correlations between observed commission rates and three model predictions (model averaged, single best and naive).

Net Error

Figure 18: Observed versus predicted rates of net error using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 18: Observed versus predicted rates of net error using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 19: Correlations between observed net error rates and three model predictions (model averaged, single best and naive).

Figure 19: Correlations between observed net error rates and three model predictions (model averaged, single best and naive).

Leave-One-Out Cross Validation

Rate Estimates

We examined the bias in estimates rates, using both the best (by AICc) model and the naive model (only random effects).

Redd Estimates

For air-based surveys, both methods provided estimates of the true number of redds that were biased high (Figure 20). However, the net error models had lower absolute and relative bias, as well as a smaller root squared mean error (RMSE) (Table 4).

Figure 20: Boxplots of absolute and relative bias for each type of predictive model.

Figure 20: Boxplots of absolute and relative bias for each type of predictive model.

Table 4: Summary statistics of predictions of total redds from leave-one-out cross validation using the net error and the omission/commission models.

Model Median # Obs. Redds Median # True Redds Median Adjustment Median Abs. Bias Median Rel. Bias (%) RMSE
Best Net Error 32 38 8.4 0.5 0.5 26.6
Best Omis / Comm Error 32 38 7.4 1.3 7.4 20.5
Naive Net Error 32 38 6.3 -0.2 -0.9 29.6
Naive Omis / Comm Error 32 38 6.8 -0.9 -3.0 25.5
Observed 32 38 - -6.0 -18.2 28.7
Figure 21: Observed number of true redds vs. leave-one-out cross validated predicted redds based on either the best AICc or naive versions of the net error or omission/commission models. Dashed line is the 1-1 line, while solid line with gray error ribbon is the best fit linear model to these data.

Figure 21: Observed number of true redds vs. leave-one-out cross validated predicted redds based on either the best AICc or naive versions of the net error or omission/commission models. Dashed line is the 1-1 line, while solid line with gray error ribbon is the best fit linear model to these data.

Discussion

References

Bates, D., Mächler, M., Bolker, B., and Walker, S. 2015. Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software, 67(1): 1–48.

Nakagawa, S., and Schielzeth, H. 2013. A general and simple method for obtaining r2 from generalized linear mixed-effects models. Methods in Ecology and Evolution 4(2): 133–142. Wiley Online Library.

R Core Team. 2019. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria.

  1. Biometrician, Biomark, Inc., ↩︎

  2. Natural Resources Specialist, Columbia Hydropower Branch at NOAA Fisheries, West Coast Region, ↩︎

  3. USDA Forest Service Rocky Mountain Research Station↩︎